Skip to content

Implement the lexer, AST and parser for the SQLite dialect - #3

Merged
kyleconroy merged 23 commits into
mainfrom
claude/the-loop-claude-md-4bnogc
Jul 30, 2026
Merged

Implement the lexer, AST and parser for the SQLite dialect#3
kyleconroy merged 23 commits into
mainfrom
claude/the-loop-claude-md-4bnogc

Conversation

@kyleconroy

@kyleconroy kyleconroy commented Jul 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Runs "the loop" from CLAUDE.md to completion, then keeps going. The parser covers the whole grammar of the pinned SQLite release, and the corpus it is checked against grew from the planned starter set to SQLite's entire test suite.

Corpus: 21,326 of 21,326 cases passing, extracted from all 945 test scripts that contain literal SQL.

The parser

  • token/ — the keyword table and %fallback ID set transcribed from tool/mkkeywordhash.c and parse.y, resolved for the pinned build's feature set (no SQLITE_ENABLE_ORDERED_SET_AGGREGATES, so WITHIN is not a keyword), plus the id/ids/idj/nm token classes.
  • lexer/ — a port of sqlite3GetToken(): four quoting styles, blob literals with even-length/hex validation, the four bind-parameter spellings including TCL-style $foo::bar(baz), _ digit separators, and the rule that 1abc is one illegal token rather than two good ones. Plus the WINDOW/OVER/FILTER resolution sqlite3RunParser does around the tokenizer. Illegal tokens stay in the stream and are reported when the parser reaches one, so tokenizer and syntax errors interleave in SQLite's order.
  • ast/ — nodes for every statement and expression form, each naming the parse.y rule it comes from, all embedding Span.
  • parser/ — recursive descent over the whole grammar. Expression precedence is a climbing loop over the eleven levels of the %left/%right block. One habit recurs and is stated once in the package doc: SQLite's LALR parser shifts a token as soon as some rule can begin with it and only then finds the rule does not continue, so where a keyword can introduce only one thing, meyer consumes it before checking what follows. Peeking would put the error on the keyword; SQLite puts it on what came next.

Conformance

Accept/reject against SQLite is the authority, with the exact message and byte offset. Three things sit under it, because accept/reject cannot see a dropped clause or a mis-associated operator:

  • A round-trip property (internal/roundtrip) — every accepting case is rendered back to SQL, re-parsed, and the trees compared structurally; rendering must also be idempotent. Used by the corpus test, the fuzz target and difftest.
  • AST snapshots — a hand-written tour of the node set under parser/testdata/ast, reviewed in diffs.
  • Span invariants — a span is a valid range, a child lies within its parent, statements are disjoint and in order. sqlc slices the source with these, and nothing else checks them.

And two searches for what the corpus does not contain:

  • FuzzParse — arbitrary bytes terminate without panicking, a rejection is a *parser.Error with an in-range offset, and anything accepted survives the round trip. 16M+ executions clean.
  • cmd/difftest — checks meyer against a live SQLite build on the verdict, the message and the offset. Its inputs are the corpus plus the ~36k strings in SQLite's fuzzdata*.db databases, which are inputs that have broken a SQLite parser at some point; each is checked directly and then mutated a token at a time (truncate, delete, duplicate, replace, insert; chained for higher orders). Over 100M mutations run clean. A weekly workflow runs ~17M of them against a fresh seed, so it actually protects the repo.

Finally, three tests read the vendored upstream sources and rebuild what meyer transcribed from them — all 147 keywords with their TK_ codes, all 73 %fallback entries, and the requirement that every one of parse.y's 133 nonterminals is named in a parser comment. Advancing the pin surfaces a grammar change as a failure.

Bugs these found

Deep nesting killed the process. meyer recurses on the goroutine stack, where exhaustion is a fatal runtime error no caller can recover from. Nesting is now counted at the three points the grammar can cycle through and rejected past 10,000 with SQLite's own "Recursion limit" wording — SQLite gives up around 2,500 and the deepest corpus case reaches 32.

ParseString panicked out of the library on an input beginning with an illegal byte: its recover was installed after newParser, which reads the first token. The fuzzer found it on its fifth seed.

NOTNULL and NOT NULL are different trees — the first is a comparison-level operator, the second takes its precedence from NOT — and the AST recorded only a bool, so the renderer wrote NOT NULL for both. 26M mutations found it; nothing else could have.

A NUL byte ends a token, not just the token stream. SQLite's tokenizer walks a NUL-terminated buffer, so an unterminated ' or [ stops at an embedded NUL and is illegal there, and a -- comment stops at one rather than at the next newline. meyer read on to the end of the Go string. Routing every scan through the cursor helper that already stood in for the C terminator fixes all of them at once. The fuzzdata corpus found this on its first run — 1,813 disagreements from the one cause.

difftest found nine more, almost all the shift-then-decide shape above: FROM t NOT NOT INDEXED reported at the first NOT instead of the second, AS NOT ( at the NOT. Plus: a BETWEEN lower bound stops at a top-level AND but absorbs a weaker operator and everything binding tighter; CREATE TABLE t(a), STRICT parses because table_option_set recurses on an empty prefix; DEFAULT 'xyzzy'. c is an error at the dot; and a register reference errors from a grammar action, which does not stop SQLite at once. The span test found a named constraint whose span excluded its own CONSTRAINT clause.

Corpus and tooling

  • The oracle records pzTail. SQLite can stop short of the end of a statement two ways, and either leaves text it never parsed. A grammar action can fail part-way through — sqlite3BeginTrigger raising no such table before the trigger body is looked at — and sqlite3RunParser abandons the rest; the corpus was demanding meyer accept three trigger bodies containing a deliberate syntax error SQLite never saw. And sqlite3_complete, which splits the script, is cruder than the tokenizer, so it can hand prepare two statements at once and only the first gets parsed. pzTail marks exactly which bytes went unverified in both cases.
  • The oracle no longer decides for itself whether a chunk holds a statement. It had a small whitespace-and-comments scanner for that, which is a second, cruder tokenizer, and it got the quirks wrong — a lone vertical tab is CC_ILLEGAL rather than space, and a /* with nothing after the star is TK_SLASH TK_STAR rather than an unterminated comment. Both showed up as phantom disagreements. prepare answers the question exactly, by returning a NULL statement.
  • One fuzz input contained PRAGMA hard_heap_limit=6874. Pragmas run at prepare time and that one is process-global, so it starved every later connection in the oracle process and the batch run died with cannot open :memory: on a flat heap. The heap limits are reset between records.
  • Extraction covers every script in the pinned tree, and do_eqp_test as well as do_execsql_test/do_catchsql_test: 4.4 MB, no curation to redo when the pin advances.
  • internal/sqlitesrc owns the pin and a batched oracle runner. Full regeneration takes ~2.5s rather than forking a process per script. Batch mode opens a fresh connection per script — reusing one leaks PRAGMA writable_schema across scripts, which silently changed five corpus files before it was caught.

Not done

The optional sqllogictest smoke gate, and milestone 8 (the sqlc integration, which lives in the sqlc repo). PLAN.md records why the sqllogictest gate is not being built: everything else the tests download is pinned by SHA-256, and sqllogictest has no stable release artifact to pin.

claude added 23 commits July 30, 2026 19:19
@claude
Implement the lexer, AST and parser for the SQLite dialect
6af55df 
Fills in milestones 1 and 3-6 of PLAN.md: the corpus harness goes from
0/4685 to 4681/4685 cases passing.

- token: the full keyword table and %fallback set transcribed from
  tool/mkkeywordhash.c and parse.y, resolved for the pinned build's
  feature set (no SQLITE_ENABLE_ORDERED_SET_AGGREGATES, so WITHIN is not
  a keyword), plus the id/ids/idj/nm token classes.
- lexer: a port of sqlite3GetToken() including the four quoting styles,
  blob literals, the four bind-parameter spellings, digit separators and
  the "1abc is one bad token" rule, together with the WINDOW/OVER/FILTER
  keyword resolution sqlite3RunParser does around it. Illegal tokens stay
  in the stream so the parser reports them in SQLite's order.
- ast: nodes for every statement and expression form, each carrying the
  parse.y rule it comes from, all embedding Span.
- parser: recursive descent over the whole grammar. Expression precedence
  is a climbing loop over the eleven levels of the %left/%right block;
  the LALR ambiguity resolutions (ON binding to a join, LIMIT x,y operand
  swap, the keyword-versus-identifier decisions that %fallback makes
  implicitly) are spelled out with comments.

The pinned build defines neither SQLITE_ENABLE_UPDATE_DELETE_LIMIT nor
SQLITE_UDL_CAPABLE_PARSER, so UPDATE and DELETE take no ORDER BY or
LIMIT; the oracle confirms both are plain syntax errors.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JzBeCg7rjweVW3uGPg5G7T
@claude
Record how far SQLite's parser got, so truncated parses do not lie
d4bf2d8 
The corpus derived "meyer must accept the whole case" from a statement
that failed with a semantic message, on the reasoning that it must have
parsed to get that far. That is wrong when the message comes from a
grammar action in the middle of a statement: sqlite3BeginTrigger raises
"no such table" at the trigger_decl reduce, sqlite3RunParser sees
pParse->rc set and abandons the loop, and the trigger body is never
parsed at all. Three such cases have a deliberate syntax error in the
body that SQLite never saw, and the corpus was demanding meyer accept
them.

sqlite3_prepare_v2's pzTail says exactly how far the parser got: the end
of the statement for an ordinary end-of-parse failure, and just past
BEGIN for these. The oracle now records it on every failing statement,
(*Case).Expected turns short tails into unverified byte ranges, and the
harness lets the parser under test fail inside one. 192 of the 4685
cases have such a range; 4 of them were failing.

Regenerated the corpus for the added field: the diff is confined to err
lines, which gain one integer each.

Corpus: 4685/4685 cases passing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JzBeCg7rjweVW3uGPg5G7T
@claude
Extract the corpus from every pinned test script, and assert error of…
6f3292a 
…fsets

PLAN.md milestone 2 named a hand-picked starter set of 69 scripts. Taking
the whole pinned test suite instead costs 4.4 MB and raises the corpus
from 4,685 to 20,971 cases across 943 scripts, with no curation to redo
when the pin advances. Scripts that yield no literal-SQL block (the
C-level tokenizer tests, the VFS and WAL harnesses) get no corpus file.

The harness now also checks the error byte offset against
sqlite3_error_offset wherever SQLite reported one, not just the message.

Two fixes this shook out of the tooling:

- An error message can span lines, because the offending token can:
  "SELECT X'0102, 1" reports the rest of the input, newline included, as
  an unrecognized token. runOracle joined nothing and failed the whole
  run; it now folds continuation lines back into the message so
  testfile.CheckCase can drop the case the way it already documents.
- regenerate-parse wrote an empty corpus file for a script with nothing
  extractable, and left a stale one behind if a script stopped yielding
  cases.

Corpus: 20971/20971 cases passing, and the parser needed no changes to
get there.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JzBeCg7rjweVW3uGPg5G7T
@claude
Add the SQL renderer, AST dump, round-trip property and snapshots
dc443db 
The second half of PLAN.md's answer to "SQLite has no parse-tree oracle".
Accept/reject conformance cannot see a dropped clause or an operator
associated the wrong way, so two layers stand in for upstream goldens:

- ast.String/ast.Statements render a tree back to SQL. Not a formatter,
  and no promises beyond re-parseability. Parenthesisation needs no
  precedence logic, because the parser keeps explicit parentheses as
  ParenExpr nodes.
- internal/dump renders a node reflectively, so it cannot fall behind the
  node set: a field added to a node shows up in the next snapshot diff
  whether or not anyone remembered. Two option sets — everything, for
  snapshots, and shape-only for comparisons, which drops the spans and
  Raw text a round trip is entitled to change.
- TestRoundTrip runs parse/render/parse over all 20,815 accepting corpus
  cases and compares the trees, then checks that rendering is idempotent.
- TestASTSnapshots covers a hand-written tour of the node set under
  parser/testdata/ast, with -update to rewrite the goldens.

Two things this found:

- IndexedColumn recorded that a COLLATE was present but not which
  collation, so "FOREIGN KEY(b COLLATE nocase DESC)" lost the name.
- Three nodes built their Span from a composite literal that also called
  the parse function for their last child, so the span ended before the
  child began. Go does not order those evaluations.

Enum types gained String methods, so a snapshot diff names what changed
instead of printing an integer.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JzBeCg7rjweVW3uGPg5G7T
@claude
Harden the parser: recursion limit, fuzzing, benchmarks, debug-parse
f8c5c41 
PLAN.md milestone 7. Two real bugs came out of it.

Deeply nested input killed the process. meyer recurses on the goroutine
stack, where exhaustion is a fatal runtime error no caller can recover
from, so "SELECT ((((...))))" with two million parens took the program
down rather than returning an error. The parser now counts nesting at
the three points the grammar can cycle through — expressions, selects
and FROM items — and rejects past 10,000 with SQLite's own "Recursion
limit" wording. SQLite gives up at a nesting of roughly 2,500 and the
deepest case in the whole corpus reaches 32, so the guard only fires on
input SQLite rejects too.

ParseString installed its recover *after* calling newParser, which reads
the first token. An input beginning with an illegal token — "\v", say —
therefore panicked out of the library instead of returning an error. The
fuzzer found this on its fifth seed.

- FuzzParse asserts three invariants over arbitrary bytes: parsing
  terminates without panicking, a rejection is always a *parser.Error
  with an in-range offset, and anything accepted survives the round
  trip. Seeded with adversarial inputs and a sample of the corpus.
  10M executions clean.
- TestRecursionLimit pins the nesting behaviour, including that a long
  flat expression is not nesting and must still be accepted.
- lexer tests cover the lexical detail accept/reject conformance cannot
  see: quoting, blob literals, digit separators, bind-parameter
  spellings, the vertical-tab quirk in aiClass[], and the
  WINDOW/OVER/FILTER resolution. Checked against the oracle.
- bench_test.go covers lex, parse and render.
- cmd/debug-parse prints the tree, the tokens or the rendered SQL, and
  points a caret at a parse error.

token.Lookup now upper-cases into a stack array instead of calling
strings.ToUpper, which it did once per identifier token.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JzBeCg7rjweVW3uGPg5G7T
@claude
Vendor the reference grammar, give errors a position, refresh the docs
0fb8a59 
- internal/reference holds upstream parse.y and tokenize.c so a comment
  naming a grammar rule can be read without a SQLite checkout. The
  directory has no Go file on purpose: nothing in the repo processes
  them, and "./..." cannot reach a directory that is not a package.
- parser.Error gains the SQL it failed on, so Error() can render
  "line:column: message" the way editors expect. Message still holds
  SQLite's wording byte for byte and is what the corpus compares; the
  exported LineCol does the conversion, which is the whole reason spans
  are stored as byte offsets.
- README covers usage, the three layers of conformance and the current
  status; PLAN.md's milestone 2 records that the corpus ended up as the
  whole pinned test suite rather than a starter set.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JzBeCg7rjweVW3uGPg5G7T
@claude
Add cmd/difftest, and fix the nine divergences it found
ce42ae7 
The corpus is whatever SQLite's test suite happens to contain, which is
overwhelmingly valid SQL: fewer than three hundred of its 20,971 cases
are rejections, so error fidelity was barely being tested. difftest
mutates corpus SQL one token at a time — truncate, delete, duplicate,
replace, insert — and checks meyer against a live SQLite build on the
verdict, the message and the byte offset. A million mutations take about
a minute.

Parser bugs it found, all the same shape in the end: SQLite's LALR
parser shifts a token as soon as *some* rule can start with it, and only
then discovers the rule does not continue, so the error names a later
token than a predictive parser would pick.

- NOT before INDEXED, DEFERRABLE or MATERIALIZED was only consumed after
  peeking at what followed, so "FROM t NOT NOT INDEXED" was reported at
  the first NOT rather than the second, and "AS NOT (" at the NOT.
- A BETWEEN lower bound stops at a top-level AND but absorbs a weaker
  operator, and everything that binds tighter than that: "x BETWEEN a OR
  b AND c" reads all of "a OR (b AND c)" and then still wants an AND.
- A comma between table constraints must be followed by one.
- table_option_set recurses on a possibly empty prefix, so a table may
  have a leading comma before its options: "CREATE TABLE t(a), STRICT".
- DEFAULT takes a bare term, so "DEFAULT 'xyzzy'. c" is a syntax error
  at the dot, not a qualified reference.
- A register reference errors from a grammar action, which does not stop
  SQLite's parser at once: it finishes the token it was handed. A syntax
  error at that same token wins; one at the next never happens.

Two harness fixes:

- The syntax-family patterns missed a multi-line message, which happens
  when the offending token spans lines (an unterminated string runs to
  the end of the input). Those were being read as semantic failures.
- The unreached ranges now also apply when the expectation is a
  rejection: a parser tripping over text SQLite abandoned never reaches
  the statement that carries the expected message.

Also here: internal/sqlitesrc owns the pinned release and a batched
oracle runner, so regenerate-parse and difftest share one pin and one
process instead of forking per script — full regeneration drops to ~19s,
and the corpus is byte-identical. Batch mode opens a fresh connection
per script; reusing one leaks PRAGMA writable_schema across scripts,
which changed five corpus files before it was caught.

parser/errors_test.go carries the regressions, with every expectation
taken from the oracle rather than written by hand.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JzBeCg7rjweVW3uGPg5G7T
@claude
Teach difftest second-order edits and the round trip, and fix trans_opt
202f1dc 
Three things difftest could not reach before: statements only the dense
snapshot inputs contain, shapes that need two edits rather than one, and
renderer bugs. Accepted mutations now go through the round trip too, so
every mutation that still parses is a free renderer test.

It found one immediately. "trans_opt ::= TRANSACTION" has no name, so a
nil Name could not say whether the keyword was written, and
"ROLLBACK TRANSACTION TO SAVEPOINT s" rendered as "ROLLBACK TO SAVEPOINT
s"... which then re-parsed with SAVEPOINT as the savepoint_opt keyword
rather than the name, and failed. BEGIN, COMMIT and ROLLBACK now record
the bare TRANSACTION keyword, and ROLLBACK TO and RELEASE record the
optional SAVEPOINT one.

1.2M mutations at depth 2, clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JzBeCg7rjweVW3uGPg5G7T
@claude
Check span invariants over the corpus, and fix named constraints
3c33d7b 
Nothing verified byte spans: a wrong one survives the corpus and the
round trip alike, and sqlc slices the source with them. TestSpans walks
every parsed corpus case and asserts that a span is a valid range, that
a child lies within its parent, and that statements are disjoint and in
source order — the last is what makes the text between two statements,
where "-- name:" comments live, unambiguous.

It found that a named constraint's span started after its own CONSTRAINT
clause, so the name was outside the node holding it. That is 20,815
cases checked, and one class of bug.

difftest's split filter also needed two rounds: sqlite3_complete's
scanner is much cruder than the tokenizer, so a ";" ends a statement for
it inside brackets, inside a bind parameter's TCL-style suffix, and
inside a string it pairs differently. Those are all incomparable, not
wrong. 22M mutations at depth 2 now run clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JzBeCg7rjweVW3uGPg5G7T
@claude
Extract do_eqp_test cases too
77d9b90 
396 invocations across the suite, with the same shape as
do_execsql_test: name, then the SQL as a brace literal. They are the
query-planner tests, so the SELECTs are among the most involved in the
suite -- multi-index OR terms, LEFT JOINs with expressions in the
constraint, correlated subqueries.

Corpus: 21326/21326 cases passing. All 355 new cases passed without a
parser change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JzBeCg7rjweVW3uGPg5G7T
@claude
Stop allocating per token and per operator
c942d5c 
The lexer built a closure over the token offset for every token, so
sqlite3GetToken's habit of reading past the token -- which relies on the
C string's NUL terminator -- cost an allocation each time. It is a
method on a small value now, and the token slice is presized: lexing a
join goes from eight allocations to two.

parseOperators built a map literal on every relational, bitwise and
multiplicative operator it consumed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JzBeCg7rjweVW3uGPg5G7T
@claude
Show enums whose zero value means something, and chain difftest edits
1fb8b3e 
A dump omits zero-valued fields, which made a snapshot ambiguous
wherever the zero is the answer: UNBOUNDED PRECEDING, PRIMARY KEY, the
NULL literal and four others were printing as nothing at all. An enum
whose zero means "absent" says so in its String -- "none", "default",
"unspecified" -- which turns out to be exactly the right test. Seven
lines gained across the snapshots, no noise added.

difftest's -depth applied every extra round to the original mutation
rather than to the previous one, so it never got past two edits.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JzBeCg7rjweVW3uGPg5G7T
@claude
Test ast traversal and the dump renderer directly
99f61c1 
Walk and internal/dump were exercised only through the parser, which
hides the cases that matter for reflection-driven code: a typed nil in an
optional field, an enum whose zero value names a real thing, and the
difference between the two Options sets.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JzBeCg7rjweVW3uGPg5G7T
@claude
Run difftest weekly
702b77d 
The tool only protects the repo if it runs. It cannot join the ordinary
CI job, which must not depend on downloading and compiling SQLite, so it
gets a workflow of its own: weekly, on demand, with the artifacts cached
against the pin.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JzBeCg7rjweVW3uGPg5G7T
@claude
Distinguish NOTNULL from NOT NULL, and start on the review findings
42c7361 
"NOT x NOTNULL" and "NOT x NOT NULL" are different trees: NOTNULL is a
comparison-level operator while "expr NOT NULL" takes its precedence from
NOT, so the first is NOT applied to a null test and the second is a null
test applied to a NOT. NullCheckExpr recorded only a bool, so the
renderer wrote NOT NULL for both and the round trip changed the tree.
26 million mutations found it; nothing else could have.

Also from the review, the findings that stand on their own:

- The oracle's "stmt ..." line format had two decoders, in testfile and
  in sqlitesrc. testfile owns the format and now exports the parser.
- ast.Stmt requires SetSpan, so the parser widening a statement's span
  is checked at compile time rather than by an assertion that silently
  does nothing.
- token.IsPlainID names the bare ID terminal that "generated ::= LP expr
  RP ID" uses, instead of assembling the class at the call site.
- parsePragmaValue was a second copy of parseSignedNumber; the lexer
  tests hand-rolled slices.Equal; the parser tests had two
  first-differing-line printers; regenerate-parse had a function-typed
  parameter that existed only to carry a string.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JzBeCg7rjweVW3uGPg5G7T
@claude
Put the round trip, the split hazard and the truncation rule in one p…
39dadb4 
…lace

Three copies of the round-trip property had already drifted: only the
corpus test still checked that rendering is idempotent, so the fuzz
target and difftest were quietly running a weaker property.
internal/roundtrip states it once and all three call it, which also
means difftest stops parsing every accepted mutation twice.

Two pieces of harness knowledge moved to where they belong:

- "the last statement failed on its own last token, so SQLite never
  reached the end of the input and cannot report the truncation" is
  expectation policy, not a difftest quirk. It sits on Expectation now,
  next to the unreached ranges it is a cousin of.
- sqlite3_complete's cruder scanner cuts a script into statements
  differently from the tokenizer, and every caller of Runner.Run
  inherits that. SplitDiffers lives beside the runner now.

Also: one corpus-walking helper for the four tests that walk it, and the
harness calls ParseString instead of routing 21,000 cases through a
reader and a per-case context that Parse documents as unused. The
io.Reader entry point gets a test of its own instead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JzBeCg7rjweVW3uGPg5G7T
@claude
Apply the rest of the review: fewer allocations, less repetition
b2ff013 
Efficiency, measured with alloc profiles over the corpus tests:

- internal/dump dominated the harness, not the parser. It rebuilt each
  node type's field layout per instance and formatted every scalar
  through fmt. The layout is cached per type and the leaves go straight
  to the builder; TestRoundTrip drops from 1.5s to 0.55s.
- regenerate-parse started and tore down a pool of oracle processes per
  script, forking a few thousand times over a run. One pool for the
  whole run takes regeneration from 19s to 2.5s, corpus byte-identical.
- parseTypeToken built a slice and joined it for every column type, when
  a type is almost always one word.
- The %fallback set and the infix operator table are Kind-indexed arrays
  rather than maps; both are consulted per token.
- Children() allocates only when a node has children, and the parser's
  named-parameter map only when there is a named parameter.

Repetition:

- parseOperators stated the precedence guard seventeen times, once per
  case, with half the operators tabled and half inline. One table, one
  guard; the operators with syntax of their own keep their cases.
- The trigger body re-implemented UPDATE, INSERT and DELETE. It shares
  their cores now, so the three real differences -- no WITH, where_opt
  instead of where_opt_ret, no DEFAULT VALUES -- are visible lines
  rather than absences.
- CreateTableStmt held Options and WithoutOpt as index-parallel slices,
  an invariant only the parser knew and only the renderer could break.
- 27 Children() methods spelled out the same append loop; writeOnConflict
  and writeOrConflict differed only in a keyword.

And the shift-then-decide rule that five comments were each re-arguing
is now stated once in the package doc.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JzBeCg7rjweVW3uGPg5G7T
@claude
Check the transcriptions against the vendored upstream sources
54cabdb 
The keyword table, the %fallback set and "every parse function names its
rule" were all things only a careful reading could confirm, and the sort
of thing that quietly rots when the pin advances. Three tests now check
them mechanically:

- token rebuilds the keyword table from mkkeywordhash.c (now vendored
  alongside parse.y and tokenize.c) and compares all 147 entries and
  their TK_ codes;
- token rebuilds the %fallback set from parse.y and compares all 73,
  both resolved for the pinned build's feature flags;
- parser checks that every one of parse.y's 133 nonterminals is named
  somewhere in the parser, which is the repository rule read from the
  side that catches an omission.

All three passed on the first run, which is the answer I wanted but not
the one worth relying on next time.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JzBeCg7rjweVW3uGPg5G7T
@claude
Feed SQLite's own fuzz corpus to difftest, and stop at NUL like SQLit…
0fed646 
…e does

PLAN.md lists test/fuzzdata*.db -- ~36k SQL strings that have broken a
SQLite parser at some point -- as a robustness corpus to use as seeds
rather than vendor. This wires them in: the oracle gains a -seeds mode
that reads one database's xsql table and frames the rows the way -batch
already expects, TestScripts unpacks the databases alongside the .test
scripts, and difftest feeds each seed in as an input of its own as well
as mutating it. That is 57k inputs and 259k checks per run.

They found four things.

A NUL byte ends a token, not just the token stream. SQLite's tokenizer
walks a NUL-terminated buffer, so an unterminated "'" or "[" stops at an
embedded NUL and is illegal there, and a "--" comment stops at one
rather than at the next newline. meyer read to the end of the Go string
instead. Routing every scan through cursor.at, which already stood in
for the C terminator, fixes all of them at once and drops the bounds
checks.

A "PRAGMA hard_heap_limit" in an input set a process-global limit at
prepare time and starved every later connection in the oracle: the batch
run died with "cannot open :memory:" on a flat heap. The heap limits are
now reset between records.

The oracle's onlyWhitespace was a second, cruder tokenizer deciding
whether a chunk held a statement, and it got the quirks wrong -- a lone
vertical tab is CC_ILLEGAL rather than space, and a "/*" with nothing
after the star is TK_SLASH TK_STAR rather than a comment. Both showed up
as phantom disagreements. Deleted: prepare answers the question exactly,
by returning a NULL statement. The corpus loses five phantom "ok" lines
for bare ";" chunks.

sqlite3_complete splits more crudely than the tokenizer, so it can hand
prepare two statements at once; prepare parses the first and leaves the
rest unexamined. "ok" lines now carry pzTail when that happens, so the
harness knows the text was never verified. No corpus case is affected.

Also: cache the oracle binary under a digest of its source, so editing
oracle.c rebuilds it, and print difftest reproducers untrimmed and
quoted -- a trimmed one hid the vertical tab that caused it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JzBeCg7rjweVW3uGPg5G7T
@claude
Bring PLAN.md's layout and hardening milestone up to date
8759bef 
The tree predates internal/roundtrip, internal/sqlitesrc, cmd/difftest and
half the parser's test files, and still calls the generator cmd/regenerate.
Milestone 7 and the robustness-corpus bullet now say where the fuzzdata
seeds actually went and why: an oracle beats "does not panic".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JzBeCg7rjweVW3uGPg5G7T
@claude
Record the fuzzdata seeds in the README, and the sqllogictest decision
6484e8f 
The one remaining item of milestone 7 is not being built, so PLAN.md should
say why rather than leave it looking pending: everything the tests download
is pinned by SHA-256, and sqllogictest has no stable release artifact to
pin.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JzBeCg7rjweVW3uGPg5G7T
@claude
Give the weekly difftest a fresh seed and a bigger budget
8ec90f8 
A fixed seed makes a weekly job re-check the same mutations forever, so it
defaults to the run id now. And the budget was set before the run was
measured: 3M checks take a minute and a half on four cores, so -per 60
-depth 3 is a few minutes for roughly 17M.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JzBeCg7rjweVW3uGPg5G7T
@claude
Document the oracle's -seeds mode where the other modes are documented
b187b20 
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JzBeCg7rjweVW3uGPg5G7T
@kyleconroy
kyleconroy marked this pull request as ready for review July 30, 2026 22:59
@kyleconroy
kyleconroy merged commit 1fa1a96 into main Jul 30, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@kyleconroy @claude